Users have long used a keyboard, mouse, trackpad, or touch screen to control an electronic device with a display screen, e.g., desktop PC, netbook, tablet, smart phone, etc. However it can be more natural for the user to control such a device using gestures, perhaps hand movements made in proximity to the display associated with the device. Such gestures require little practice compared to mastering a keyboard, and do not require the user to look away from what might be displayed by the device. Further there is less repetitive motion, which can decrease likelihood of repetitive stress injury (RSI) to the user.
Systems and methods to detect and recognize gestures, including three-dimensional gestures, can vary from the highly complex and expensive to the relatively straightforward and inexpensive. However implemented, such systems and methods seek to enable a user to control an electronic device including controlling what appears on the device display screen, preferably without necessarily touching the electronic device. The display screen may but is not required to be an integral portion of the device housing. While user touching a touch screen device is a common mode of scrolling text, selecting menu items, manipulating graphics and the like, there are several disadvantages to user interactions that rely upon a touch screen. Touching tends to smudge the device touch screen with fingerprints, and requires the user to literally be within an arm's length of the touch screen surface. Touch screens simply are not feasible for a large sized device, perhaps a smart TV, that may be wall mounted. This problem is aggravated when user view streaming video and the like over such large screen devices in that they tend to sit some distance away from the screen to enjoy the presentation. Even if they could do so, the user would have to stand up and approach the smart TV to interact by touching its display screen. Of course a user could employ a remote control but that would require holding another device. Portable devices such as tablets may have touch screens and often have flimsy stands to prop-up the device for ease of user touching. But it is too easy for a user to accidently tip such devices over when interacting by touching the screen. Handheld devices such as smart phones have relatively tiny display screens, and requiring a user to touch the screen surface physically restricts the area of user interaction to a relatively small region. Just as repetitive keyboard typing is known to cause stress injury, it is recognized that making certain types of repetitive gestures can cause injury known as ‘gorilla arm syndrome’. (See http://www.wired.com/gadgetlab/2010/10/gorilla-arm-multitouch/.) Some devices provide for user interaction within a zone slightly spaced-apart from the display screen surface, without touching the surface. But the user is required to hold his/her hand very close to the screen surface within the screen boundaries. Such movements are not feasible for a wall mounted device, and can be difficult to accomplish for small portable devices such as smart phones.
A graphical user interface (GUI) is a favored mode of user-device interaction and refers to user selection of objects presented on the device display screen. Examples of selectable GUI objects may include icons, buttons, check boxes, menus and menu items, spreadsheet cells, location in a GUI canvas such as a location of a displayed map, a weapon trigger in a displayed video game, etc. The term selection refers to the act of selecting a selectable GUI object. The term “detection” refers to the software code that detects and maps the user object to a predefined coordinate system. Detection further identifies the target item with which the user intends to interact. The user's intent, what the user wants to do with the selected object, may be defined as part of the selection or by the next user action. For instance, user selection of a displayed menu item may open up a list of sub-menus, or a so-called ribbon. It is known to enable the user to select an edge or corner of the display whereat no object need be displayed to reveal hidden menus that can provide additional user options. If the displayed object is perhaps a geographic map, user selection of a map location can signal to the device and its detection system a ‘grab’ function. Subsequent user created motion of the map selected point can scroll the map in the direction of the motion.
Gestures are a form of human communication using body parts, e.g., user fingers, hands, face, head, shoulders, torso, feet, toes, etc. Gestures are called natural interfaces because they can be performed without use of any additional item, e.g., no stylus, no special gloves, etc. need be used. Gestures can also input data to computers and thus augment other forms of natural interfaces such as speech.
User gestures occur in a real world situation, and their recognition requires a reliable method of three-dimensional input so they can be detected and parsed for identification using computer software. When a gesture is used to communicate with a computer, higher vocabulary of gesture types or gesture language may be defined. For instance, a clockwise or counter-clockwise user hand rotation gesture may be used to mimic increasing or decreasing volume of a virtual rotary knob perhaps displayed on the device screen.
How a gesture is interpreted can be customized to various applications on the computer device. For instance, a waving type gesture can be used to get the attention of the computer device to identify a user about to communicate with the computer device. The computer device consequently can enter a mode to accept subsequent user gestures. A gesture may be defined to start a specific application, or even to unlock or lock the computer device.
Certain gestures are known in the prior art, for example “pinching” or “expansion” and “pausing”, as are use of gestures for three-dimensional guided navigation, as in motioning left or right with a user finger. TABLE 1 below represents a survey undertaken by applicant of prior art gestures. In Table 1, ten digit references are pending U.S. patent applications. Issued U.S. patent numbers are prefaced with the abbreviation USP. Applicant's recital of the below references is not to be construed as an admission by applicant that the references should indeed be patentable or are necessarily relevant to aspects of the present invention described here. The references are merely cited as a broad survey of what the prior art generally seems to encompass relative to gestures.
TABLE 1TaxonomyReferenceDisclosure3D gestures - curtain2012/0274550 “Gesture Mapping for 3D hand gesture moves laterally to cause application toDisplay Device” (Campbell, 2010)turn reading material page [0022, FIG. 105A];entire hand vertically is similar to vertical curtain [0026,FIG. 106B]3D gestures - pinch2012/0274550 “Gesture Mapping for3D pinching using thumb and forefinger pinched together,and zoomDisplay Device” (Campbell, 2010)pinching and then rotating hands to rotate object [0027]3D gestures - pinch & U.S. Pat. No. 8,086,971 “Apparatus, acoustic sensing - zoom in/zoom out by expanding/zoomMethods and Computer Program closing two fingers including the thumb (FIGS. 5A, 5B, 8D,Products Providing Finger-Based and p. 6 line 20)Hand-Based Gesture Command” (Nokia, 2006) . . .3D gestures - pinch2012/0204133 “Gesture-Based Userperforming grab gesture by pinching thumb with two orInterface” (Primesense 2012)more fingers [0061, FIG. 6B]3D gestures - pausing U.S. Pat. No. 7,665,041 “Architecture pausing full hand movement to interact with the systemfor Controlling a Computer Using and perform tasks such as select, bring window toHand Gestures” (Microsoft), 2003)foreground [p 17, lines 55-65]3D gestures - pausing2009/0103780 “Hand-Gesturepoking and pausing with extended index finger Recognition Method” (Nishihara),[0048, FIG. 5]2008)3D gestures - pausingU.S. Pat. No. 8,086,971 “Apparatus, 3D pausing - finger stays motionless to select an objectMethods and Computer Program displayed on the screen (p. 9, line 3).Products Providing Finger-Based and Hand-Based Gesture Command” (Nokia, 2006) . . .3D gestures -2011/0219340 “System and Methodpointing with index finger [FIG. 1 defines a “pecking handpoking/pausingfor Point, Select and Transfer Handgesture” and is a pecking action [0027]Gesture Based User Interface”(Pathangay, 2010)3D gestures - poking8,086,971 “Apparatus, Methods Andclicking with a poking/pointing single finger or moving the(clicking)Computer Program Products Providingfinger slowly under a displayed object (p. 6, line 5), andFinger-Based and Hand-Basedsubsequent tapping by a single finger [p 8, line 65]; fingerGesture Command” (Nokia, 2009))stays above an object followed by a slow movement [p 9,line 5]3D gestures - cocking2009/0103780 “Hand-Gestureextended index finger and extended thumb, user thumb(thumb cocking)Recognition Method” (Nishihara, 2008)movement [0052, FIG. 5] (Less cocking, more movementof thumb closer to index finger)3D gestures - cocking2009/0103780 “Hand-Gesturecocking with index finger [0062]; pointing with index finger(thumb)Recognition Method” (Nishihara, 2008)[Claim 31]3D gestures - others2012/0274550 “Gesture Mapping Formoving entire hand vertically, similar to a vertical curtainDisplay Device” Campbell 2010)[0026, FIG. 106B]3D gestures - others8,086,971 “Apparatus, Methods andmultiple 3D gestures - open closed hand, make circle withComputer Program Products Providingtwo fingers, clockwise/anticlockwise rotation with singleFinger-Based and Hand-Basedfinger [FIG. 4B], double click by two fingers, moving twoGesture Command” (Nokia - 2006)fingers. Switching on/ off based on change in direction offingers movement, or alternatively finger acceleration (p. 6lines 0-20)(p 8-9)3D gestures - othersU.S. Pat. No. 8,96,114 “Gesture virtual dry-erase board function on which users can drawRecognition Interface System With or write [p 5, line 40]; user must touch display screen withVertical Display” (Northrop, 2008the index finger only, or with index and thumb finger, orpalm of hand to initiate the dry erase board in differentcontexts. [p. 11, lines 50-70, p. 12]3D gestures - othersU.S. Pat. No. 7,665,041 “Architecture uses both 3D gestures and voice to interact withfor Controlling a Computer Using computer [p 17, lines 40-50]; bringing fingers together withHand Gestures” - (Microsoft, 2003thumb and subsequently moving full hand to bring awindow to the forefront [p 18 lines 5-15]; moving fingersvertically [p 18, lines 20-35]; has “record gesture mode”for users to record new gestures [p 18, lines 45-55]; alsouses voice recognition to interact with gestures [Claim 24]3D gestures - others2009/0103780 “Hand-Gestureboth single hand and double hand gestures pointing bothRecognition Method” index fingers [0047, 0056, FIG. 6]; flat extended hand, with(Nishihara, 2008)fingers together or spread out [FIG. 5, 0049]; contact withthe display surface with at least one finger - index fingerand thumb, two fingers [0050, FIG. 5; opening all fingersof closed hand [FIG. 7]; cocking with index finger [0062];well described library of gestures3D gestures - othersU.S. Pat. No. 7,340,077 “Gesture rich hand gestures, such as directing one hand at anRecognition System Using Depth electronic device while raising two fingers [p 12, line 20]Perceptive Sensors” (Gokturk, 2003))3D gestures - othersU.S. Pat. No. 8,96,114 “Gesture touching a vertical surface with a raised index finger andRecognition Interface System With a closed fist [p 11, line 60]; touching the surface with aVertical Display” (Northrop, 2008)flat palm [p 12, line 60]3D gestures others2012/0204133 “Gesture-Based Usergrab (closing hand to make a fist or folding one or moreInterface” (Primesense, 2012)fingers towards palm) [0060, FIG. 6A]; release (openinghand from closed or folded state to release Grab) [0062,FIG. 6C]; push/pull (moving hand towards an applicationobject or moving hand away) [0063, FIG. 7]; rotating palmclockwise/counter clockwise [0065, FIG. 8A, 8B]3D gestures - others2011/0219340 “System and Methodgrabbing or half grabbing with full hand either closed orfor Point, Select and Transfer Handopen [0042-0043]Gesture Based User Interface”(Pathangay, 2010)3D gestures - guided2010/0199228 “Gesture Keyboarding”can use 3D gestures to type onto a displayed virtual(based on menu)(Microsoft, 2009)keyboard [0093, FIG. 6]3D gestures guidedU.S. Pat. No. 7,665,041 “Architecture hand gestures point to, expand/close menus [0061]for Controlling a Computer Using HandGestures” (Microsoft), Mar. 25, 20033D gestures guidedU.S. Pat. No. 8,086,971 “Apparatus, one finger stays above an object, then an object-specificMethods and Computer Program menu appears, the other finger performs circular rotationsProducts Providing Finger-Based and and toggles through menu [p 9, line 35]Hand-Based Gesture Command” (Nokia, 2006)entering 3D InteractionU.S. Pat. No. 8,96,114 “Gesture option to enter into virtual dry erase white board mode byzone (waving)Recognition Interface System With touching display screen with one finger, with hand mostlyVertical Display (Northrop, 2008)closed; application then interprets future gestures as tobe displayed in real time on virtual dry erase board. [p 11,lines 55-65]entering 3D interactionU.S. Pat. No. 7,665,041 “Architecture can trigger system attention by making specific gesture,zone (waving)for Controlling a Computer Using Hande.g. raising hand [p 21, lines 5]; discloses rich gestureGestures” (Microsoft, 2003)library (See FIGS. 7, 8] 3D gesturesU.S. Pat. No. 7,340,077 “Gesture specific hand gestures used as deliminators - to indicatedeliminatorsRecognition System Using Depth beginning or end of a hand gesture. [p 12, lines 40-45]Perceptive Sensors” (Gokturk, 2003)coupling 3D object hotU.S. Pat. No. 8,232,990 “Working with method to modify 3D object on touch screen based on apoints with fingertips3D Objects” (Apple, 2010)3D gesture [Claim 7, FIG. 2C]; NB: method requiresfingertips to first start with contact with touch screensurface and then lift off surface to modify 3D object.enriched gestureU.S. Pat. No. 7,665,041 “Architecture claims tracking gesture characteristics including at leastlanguage using otherfor Controlling a Computer Using one of hand movement, finger orientation, finger count,properties of 3D object Hand Gestures” (Microsoft, 2003finger orientation and hand rotation [Claim 9](velocity, fingertiprotation, etc.)enriched gestureU.S. Pat. No. 8,086,971 “Apparatus, uses fingertip landmarks, velocity, and rotation of thelanguage using otherMethods and Computer Program fingertips. [page 10, lines 20-30, page 11, lines 45-50]properties of 3D objectProducts Providing Finger-Based (velocity, fingertipand Hand-Based Gesture Command” rotation, etc.)(Nokia, 2006)enriched gestureU.S. Pat. No. 7,665,041 “Architecture flat open hand that can roll in six degrees of motion for alanguage using otherfor Controlling a Computer Using rich gesture library [0051, 0054]properties of 3D objectHand Gestures” (Microsoft, 2003(velocity, fingertiprotation, etc.)enriched gestureU.S. Pat. No. 7,340,077 “Gesture while pushing two fingers on display screen, depthlanguage using otherRecognition System Using Depth information re hand movement used to process twoproperties of 3D objectPerceptive Sensors” (Gokturk, 2003separate commands, although fingers pushed(velocity, fingertipsimultaneously. [p 15, lines 20-30]; can also userotation etc.)fingernails on screen rather than full finger contact todraw a thin line in a virtual canvas. [p 16, lines 30-35]enriched gesture2012/0204133 “Gesture-Based Usercontext specific applications: hand fast down movementlanguage using otherInterface” (Primesense, 2012)interpreted as “mute” button, rather than volumeproperties of 3D objectdecrease on a sound application. [0056](velocity, fingertiprotation etc.)Z0 planeU.S. Pat. No. 7,665,041 “Architecture has moveable engagement plane positioned so user canfor Controlling a Computer Using control system by moving hand or object through theHand Gestures”(Microsoft 2003engagement plane and into engagement volume;engagement plane is beginning of engagement volume.[p 16, lines 40-50, FIGS. 9A, 9B]. When object is movedthrough engagement frame, feedback is displayed foruser; engagement plane used primarily to start interactioninto engagement volume, and not as focus point forinteraction itself.signal showing 3DU.S. Pat. No. 7,665,041 “Architecture when object moved through engagement frame,interaction zone hasfor Controlling a Computer Using feedback displayed for user [p 16, lines 40-50, FIGS. 9A,been entered (buzzer,Hand Gestures” (Microsoft, 20039B].haptic feedback, visualindicator)Smart 3D Context2010/0199228 “Gesture Keyboarding”when entering keyboard characters with gestures, systemAware Applications(Microsoft, 2009is aware that a gesture is used and can attempt character(assumes a 3Dautocorrect due to errors in recognizing gestures [0091]gesture has occurred,e.g., resizing screenand menu options)Smart 3D Context2011/0173574 “In Application gestures switch between modes on a computerAware ApplicationsGesture Interpretation” application, and can turn on an application, or control(based on the fact that(Microsoft, 2011)some hardware component. [0038, 0041]a 3D gesture hasoccurred, e.g., resizingscreen and menuoptions)
What is needed is a system and method enabling customizable recognition of many types of three-dimensional gestures such that a user can render comfortably and without creating undue stress or strain in the user's arm or hand. Further, there is a need for a system and method that can provide effective user feedback such a graphical user interface (GUI) imagery, perhaps sound, and preferably including haptic feedback, commensurate with a gesture and/or before applying the gesture to the electronic device. Finally such method and system should be low in cost and in power consumption.
The present invention provides methods and systems with such features.